当神经网络失去先前从不同分布的样本(即新任务)培训一组样本时,发生灾难性遗忘(CF)。现有方法在减轻CF方面取得了显着的结果,尤其是在称为任务增量学习的情况下。但是,这种情况是不现实的,并且已经完成了有限的工作以在更现实的情况下取得良好的结果。在本文中,我们提出了一种称为Centroid匹配的新型正则化方法,该方法受到元学习方法的启发,通过在神经网络产生的功能空间中操作来打击CF,在需要较小的记忆足迹的同时,取得了良好的结果。具体而言,该方法使用神经网络产生的特征向量直接对样品进行了分类,通过将这些向量与代表当前任务中的类或所有任务的质心匹配,直到该点。质心匹配速度比竞争基线更快,并且可以通过在过去的任务结束时保留模型产生的嵌入式空间之间的距离,并且可以利用它有效地减轻CF,而当前生产的距离则可以实现高精度的方法在所有任务上,在轻松场景上操作时,或不使用外部内存,或者将小型内存用于更现实的记忆。广泛的实验表明,匹配的质心在多个数据集和方案上取得了准确的提高。
translated by 谷歌翻译
Catastrophic forgetting (CF) happens whenever a neural network overwrites past knowledge while being trained on new tasks. Common techniques to handle CF include regularization of the weights (using, e.g., their importance on past tasks), and rehearsal strategies, where the network is constantly re-trained on past data. Generative models have also been applied for the latter, in order to have endless sources of data. In this paper, we propose a novel method that combines the strengths of regularization and generative-based rehearsal approaches. Our generative model consists of a normalizing flow (NF), a probabilistic and invertible neural network, trained on the internal embeddings of the network. By keeping a single NF throughout the training process, we show that our memory overhead remains constant. In addition, exploiting the invertibility of the NF, we propose a simple approach to regularize the network's embeddings with respect to past tasks. We show that our method performs favorably with respect to state-of-the-art approaches in the literature, with bounded computational power and memory overheads.
translated by 谷歌翻译